A Greedy Divide-and-Conquer Approach to Optimizing Large Manufacturing Systems using Reinforcement Learning

Manufacturing is a challenging real-world domain for studying hierarchical MDP-based optimization algorithms. We have recently obtained very promising results using a hierarchical reinforcement learning based optimization algorithm for a 12-machine transfer line. Transfer lines model factory processes in automobile and many other product assembly plants. Unlike domains such as elevator scheduling, where the individual elevator \agents" are eeectively independent , individual machines in a transfer line are directly aaected by each others behavior. This interaction creates a highly non-stationary environment that defeats ""at" learning algorithms that train each machine while they are part of the line. Our hierarchical optimization algorithm comprises of an average-reward Q-learning algorithm for semi-Markov decision processes called SMART for learning low-cost policies for operating each individual machine, and a greedy algorithm for combining the base level policies to obtain an overall policy for running the transfer line. Unlike other approaches to hierarchical optimization, in our system policies at the lowest level are modeled using semi-Markov decision processes (SMDPs). In our results to date, we can show that the hierarchical approach is not only much faster than the ""at" algorithm, but also appears to outperform well-known heuristics for running transfer lines used in many factories today.